LAGC: Lazily Aggregated Gradient Coding for Straggler-Tolerant and Communication-Efficient Distributed Learning

Gradient-based distributed learning in parameter server (PS) computing architectures is subject to random delays due straggling worker nodes and possible communication bottlenecks between PS workers. Solutions have been recently proposed separately address these impairments based on the ideas of gradient coding (GC), grouping, adaptive selection. This article provides a unified analysis techniques terms wall-clock time, communication, computation complexity measures. Furthermore, order combine benefits GC grouping robustness stragglers with load gains selection, novel strategies, named lazily aggregated (LAGC) grouped-LAG (G-LAG), are introduced. Analysis results show that G-LAG best time performance while maintaining low computational cost, for two representative distributions times nodes.